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Abstract. Low-density parity-check codes with irregular constructions have been 
recently shown to outperform the most advanced error-correcting codes to date. In 
this paper we apply methods of statistical physics to study the typical properties of 
simple irregular codes. We use the replica method to find a phase transition which 
coincides with Shannon's coding bound when appropriate parameters are chosen. The 
decoding by belief propagation is also studied using statistical physics arguments; the 
theoretical solutions obtained are in good agreement with simulations. Wc compare 
the performance of irregular with that of regular codes and discuss the factors that 
contribute to the improvement in performance. 



PACS numbers: 89.70+c 89.90+n 05.50+q 
1. Introduction 

Error-correction mechanisms are essential for preventing loss of information in 
transmissions through noisy environments. They are of increasing technological 
importance with applications ranging from high capacity storage media to satellite 
communication. The surprising fact that error-free communication is possible if the 
information is encoded to include a minimum amount of redundancy was discovered 
by Shannon in 1948 Shannon proved that a message encoded at rates R (message 
information content/code- word length) up to the channel capacity C can be decoded 
with vanishing average error probability ^ as the length of the message increases 
M ^ oo. This theorem was then progressively refined by Gallager and others (see [Q] 
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and references therein) to say that the average over messages and codes of the error 
probabihty is bounded by 

p^^^-MEiR)^ (1) 

where E{R) is the error exponent that is greater than zero for rates up to the channel 
capacity C. 

These proofs were presented in a non-constructive form by assuming encoding 
processes by ensembles of unstructured random codes and impractical decoding methods 
like maximum likelihood or typical set decoding 0. No encoding- decoding scheme that 
is practical and attains the coding bound has been found to date. 

The most successful code in use to date is the Turbo code [^]. However, the current 
performance record is owned by an irregular low-density parity check code (LDPC), 
more specifically an irregular Gallager code ^. This code was first proposed by 
Gallager in 1962 p, 0, and were all but forgotten soon after due to technical limitations 
of the time. Recently a variation of the original proposal by Gallager named MN code 
has been proposed by MacKay and Neal H, they showed that this code has good 
performance, what attracted renewed interest to LDPCs. Since then LDPCs have been 
reconsidered in a variety of architectures [0, |TT|. Some of which reported close to 
optimal performance |T2|, |T3| . 

Representing a message by a binary vector ^ G {0, 1}^, the LDPC encoding process 
consists of producing the binary vector t G {0, 1}^^ defined hj t = G^s (mod 2), 
where all operations are performed in the field {0, 1} and are indicated by (mod 2) and 

is a M X N generator matrix. The transmission is then corrupted by noise, that 
we assume to be a binary vector ^ G {0, 1}*^, and the received vector takes the form 
r = G'^^ + C (mod 2). The decoding process is performed by applying a suitable parity- 
check matrix to the received message to produce the syndrome vector z = Ar (mod 2). 
The parity-check matrix A defines the code structure and can be represented by a 
bipartite undirected graph with check and bit nodes. This gives rise to the classification 
of LDPCs to regular (those forming regular graphs) and irregular codes. 

The parity-check matrix for Gallager codes is a concatenation A = [Ci | C2] of two 
very sparse matrices, with C2 (of dimensionality (M — A^) x (M — A^)) being invertible 
and the rectangular matrix Ci of dimensionality (M — A^) x A^. The generator matrix 
of a Gallager code is G = [7 | C^^Ci] (mod 2), where / is the N x N identity matrix, 
implying that AG^ (mod 2) = and that the message itself is set as the first A^ bits in 
the transmission. The syndrome vector is then z = Ar = A(^ (mod 2) from which the 
noise can be estimated and subtracted from the received message. For a MN code the 
generator matrix has the form G^ = C~^Cs (mod 2), where Cn is an M x M invertible 
matrix and is M x A^. The matrix applied by the decoder is given by Cn producing 

f See http://www331 .jpLnasa.gov/public/JPLtcodes.htm\ for JPL's " impcrfectness" contest. 
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z = CnV = Cs$, + CnC (mod 2), from which the most probable message vector can be 
predicted. 

Although Gallager and MN codes can be analysed by the same methods of 
information theory 0, they represent different physical systems with different 
properties. In this paper we will restrict the analysis to irregular MN codes, the analysis 
of Gallager codes will appear elsewhere. 

Statistical physics has first been applied to the analysis of error-correcting codes in 
the seminal work of Sourlas [IH which has been recently extended to the case of finite 



code rates [Ig, [16|. Similar methods have been recently applied to the case of Turbo 



codes [|T9l and regular MN codes ||T^ |l^], providing a detailed description of the system's 
phases and capabilities for various parameter choices. Here we analyse irregular MN 
codes using the standard replica calculation to find a free energy that is a measure of 
the likelihood of typical solutions to the decoding problem, given an ensemble of code 
matrices Cg and Cn {code construction) , channel and message models [noise level and 
message bias). 

We show that three types of solutions emerge depending on the parameters provided: 
successful errorless decoding (number of incorrect bits less than 0{N)), imperfect 
decoding (number of incorrect bits of order A^) and complete failure (number of correct 
bits less than 0{N)). We also show, as in |T8[, that the line separating errorless 
and complete failure phases can coincide with the coding limit; this fact itself is not 
particularly surprising as the statistical physics analysis relies on the same kind of 
arguments used in the original coding bounds, using averages over ensembles of codes 
and maximum likelihood decoding. The main difference here is that the matrices in the 
ensemble have some structure. 

The statistical physics approach can be regarded as complementary to that of 
information theory; it enables one to attain a more complete picture by analysing the 
decoding problem in the infinite message limit and by looking at global properties of the 
free energy. It allows for a transparent analysis of the possible performance of different 
codes characterised by different choices of construction parameters, and has already 
resulted in new practical high performance codes [jl3| . 

In this framework, Bayes-optimal decoding generally corresponds to finding the 
global minimum of a TAP free energy pO|, ^ which is very costly if the landscape has 
multiple local minima. A practical decoding algorithm that has been used in LDPCs 
is the scheme known as belief propagation, broadly used by the Bayesian inference 



community p6|, ^7j. Belief propagation is equivalent to solving iteratively a set of 
coupled equations for finding extrema (local or global) of the TAP free energy p!8|,[28[1. 
This method is very sensitive to the presence of local minima and can be easily trapped 
in sub-optimal solutions. 
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In this paper we study the dependence of the free energy surface on the noise level 
and the message bias; this allows us to study the solutions which exist in each one of 
the cases and to detect the emergence of suboptimal solutions that will interfere in the 
practical decoding dynamics. 

This paper is organised as follows: Section 2 presents irregular MN codes, while 
the statistical physics analysis is outlined in Section 3; the relations between the belief 
propagation approach and statistical physics are discussed in Section 4 and employed to 
examine the decoding performance in Sections 5 and 6. Concluding remarks are given 
in Section 7. 

2. Irregular MN codes 

Although the best irregular LDPCs found so far are defined in g-ary alphabets ||3^, we 
will restrict the current analysis to the binary alphabet {0, 1}. 

We suppose that the binary messages S comprise independent bits sampled from 
the prior distribution P{S) = (1 —p) 6{S) +p6{S— 1), where 6{S) stands for the Dirac's 
delta distribution. We also assume a simple memoryless Binary Symmetric Channel 
(BSC) with binary vectors r having independent components sampled from a similar 
prior distribution of the form P{t) = (1 — /) 6{t) + f 6{t — 1). From now on we will 
reserve the symbols ^ and C for the actual message and noise, using S and r for denoting 
random variables in the message and noise models. 

The goal is then to find the Bayes-optimal estimate Sj = a.Ygmaxg.TYSi^j,TP{S, r | 
z); the matrices Cn and Cs are also given, but were omitted for brevity. 

One can use Bayes formula to incorporate the prior knowledge on message and noise 
and write the adequate posterior probability: 

P(5, t\z) = ^x {CsS + Cr,T = z (mod 2)} P(5)P(t), (2) 

where the indicator function is x {^} = 1 if ^ is true and otherwise. 

The matrices are chosen at random in such a way that Cn is invertible over the field 
{0, 1} and a row m in Cs and Cn contains Km and Lm non-zero elements respectively. 
In the same way, each column j of Cs contains Cj non-zero elements and each column 
/ of Cn contains Di non-zero elements. 

Parity-checks for the signal and noise bits are specified by the matrices Cs and 
Cn respectively. The system can be mapped onto a bipartite graph represented by 
{Cs I Cn) (adjacency matrix in the graph theory jargon), to say, each one of the M 
rows lists the bit nodes connected to a check node and each one of the N + M columns 
lists the checks conveying information about the particular bit node. Therefore, the sets 
{Km}m=i and {L„}*^^ give the order of check nodes, {Cj}^^^ and {Di}f^^ the order of 
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bit nodes. Clearly this sets must obey the relations: 

N M M AI 

Y.C,= Y.Km Y.Di=Y.Ln.. (3) 

j=\ m=l 1=1 m=l 

standing for the number of edges in the signal and noise graphs respectively. 

The information rate of the code is given by i? = -f^2(p) M/N, where H2{p) = 
—p log2(p) — {1 — p) log2(l — p) is the binary entropy of the source. 

Alternatively one can write R = H2{p)K/C, where: 

-r Af N 

K = ±-Y.Km C=^^yCj (4) 



To simplify the calculations we change, as in the original work by Sourlas the 



representation of the variables, replacing the field {0, 1} by {±1} and modulo 2 sums 
by products. Moreover, we restrict our analysis to the case of irregular bit nodes (sets 
{CjjjLi and {Di\\L^^ and regular check nodes (fixed K and V). The case with regular 
bit nodes and irregular check nodes is the basis for high performance codes studied in 



3. Equilibrium theory 

To assess the performance of irregular MN codes we compute, using standard techniques, 
the free energy of the system / = — lim7v^oo;^(ln 2') where Z is the normalisation in 
(0). The average (...) is performed over the matrices C„ and C^, the messages ^ and 
the noise <^ and will provide information about the typical performance of these codes. 

In the ±1 representation, the syndrome vector z = CnV = Cs$, + CnC (mod 2) 
becomes Jf,a = UjeiM H/Ga 0, where fi = {h, ■ ■ ■ .ir) and a = {k, - ■ ■ Jl) are sets of 
indices corresponding to the non-zero elements in one of the M rows of Cg and Cn 
respectively. 

The prior distribution over the message bits Sj E {±1} becomes P{Sj) = 
(1 — p)6{Sj — 1) + pS{Sj + 1), while for the noise bits ti G {±1} one has -P(r/) = 
{l-f)6{n-l) + f6{n + l). 

The code construction is specified by the tensor Afj_a- G {0, 1} that determines the set 
of indices fia which correspond to non-zero elements in a particular row of the matrix 
{Cs I Cn). To cope with non-invertible Cn matrices one can start by considering 
an ensemble with uniformly generated M x M matrices. The non-invertible instances 
can then be made invertible by eliminating a e ~ 0(1) number of rows and columns, 
resulting in an ensemble of (M — e) x (M — e) invertible C„ matrices and (M — e) x (A^ — e) 
Cs matrices. As we are interested in the thermodynamical limit we can neglect 0{1) 
differences and compute the averages in the original space of M x M matrices. The 
averages are then performed over an ensemble of codes generated as follows: 
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(i) sets of numbers {Cj}^^ and {Di}fl^ are sampled independently from distributions 
Vc and Vd respectively; 



Cj and 



(ii) tensors A^a are generated such that X)^o- ^f^a = J2{fi:j£fi} ^fia 

J2{a:ie(7} -^Atcr = -D/, where {fi : j & fi} stands for all sets of indices that contain 
j- 

The indicator x iii (@) can be replaced by a more tractable function that is 
E{S,t;A) = 1, if the dynamical variables S and r satisfy ^J^o- = Hje^i SjHiea'^'i and 
E{S, t;A) = otherwise. This function has the form: 



£"(5, r; A) = hm exp < V Ana 



n n 



(5) 



The priors over message and noise take the form of external fields in the statistical 
physics framework and can be written in an exponential form with the normalisation 
incorporated in the partition function Z: 



(N M 
j=l 1=1 

the fields are then Fg = atanh(l — 2p) and F„ = atanh(l — 2/). 
As in W7\, IT3, the partition function becomes: 



(6) 



Z = lim Tr 



S,T 



j&l-l leer 



p N p M 



1=1 



(7) 



Performing the gauge transformation Sj ^ ^jSj and r/ C^iTi one obtains: 



M 



(8) 



o'e/j lea 



The resulting Hamiltonian represents a multi-spin ferromagnet in a random field, 
the disorder is transformed as jT^o- ^ 1 under the gauge transformation, and therefore, 
is trivial and there is no frustration in the system. The different phases that will appear 
are then due to competition between the local fields and ferromagnetic interactions. 
Due to the structure of (|^) all the thermodynamics is obtained in the zero temperature 
limit unlike the Sourlas' code case where optimal decoding must be carried out at finite 
temperatures [^, p3|, ^ p5|. 

The free energy f{p,f,a,Vc,VD) = — liniAr^oo;^(ln ^ ^ can be determined 



using the replica method along the same lines as reported in fl^, [ly, [I^], but for 



the irregular case it also depends on the probability distributions Vc and Vd used to 
generate the ensemble of codes. The auxihary variables qai-a„, = N"'^ J2j ^jS'j^ ' ' ' S""^ 



and r„j...Q^ = M ^Y^iYiTt^ ■ ■ ■ t^"" , and their conjugates qai-a,n and f„j...Q„ , emerge 
from the calculation. The replica symmetry assumption is enforced by using the ansatze: 



and 



dyp{y) y"" ^ai-a™ 



dx vf (x) x"' 
dy p{y) iT. 



(9) 

(10) 



The expression for the free energy then follows: 
f{p, f, a, Vc, Vd) = ^^t^-^^-p}[a In 2 
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a 



K 

n dxj7i{xj] 



n dyipiyi 



.1=1 



K L 

\n\l + Xlx,X{yi 

3=1 1=1 



+ C I dx Tc{x) dx Tr{x) In (1 + xx) + a L j dy p{y) dy p{y) In (1 + 

c 



c 





' c 




f 


n n{xj) 


^ In 









c 



D 



D 



n dm Pirn. 



.1=1 



In 



i=i j=i 
e^^^ f[il + y,) + e-^^^ f[{l - y,: 



1=1 



1=1 



where a = M/N = C/K. 

The system's states are obtained by the extremization above, resulting in the saddle- 
point equations : 

K-l L 



^(^) = / n dxj Ti{xj)\{dyi p{yi] 
j=i 1=1 

„ K L-l 

P(y) = / n 7r{xj) Yl dyi p{yi) 

3=1 1=1 



K-l L 

^ - n n 



j=i 1=1 

K L~l 

j=i 1=1 



(12) 



J-- 

P{y) = Y.='T'D{D) j\{dyip{y{) 1 5 

D ^ -^1=1 \ 



C-1 



X — tanh Fs^ + ^ atanh (xi) 
\ 1=1 

/ D-l ^ 

y — tanh I F^-C + E atanh (yi) 



1=1 



The exact meaning of the fields n, n, p and p were presented in ||16| , p8| and will be 
further discussed in the next section. 

Due to (|]) the estimate for the message is S = sgn((S')/3_>oo), where the average is 
thermal with Hamiltonian (§) in the zero temperature limit. The decoding performance 
can be measured by 



m 



N 



N 



i=l 



dh (j){h) sgn{h), 



(13) 
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where, as in |18 



m = T.'Pc{c) J f[dx 



J T\{Xj) ( 5 



c 



h — tanh Fg^ + ^ atanh [xi] 
\ 1=1 



(14) 



c j=l 

Solutions can be found easily for the case where Fg = (unbiased messages) and 
the code constructions are generated by distributions Pd{D) and PciC) that vanish for 
< C, D < 2 (codes with at least two checks per bit). For K,L > 2 one finds just two 
types of solutions: a ferromagnetic state with magnetization m = 1, 

7i{x) = 5[x - 1] 7f(x) = S[x - 1] (15) 
p(x) = 6[y - 1] p{y) = 6[y - 1], 

and a paramagnetic state with m = 0, 

7r(a;) = 6[x] vf (2) = 6[x] (16) 

p(x) = {S[y - tanh(CF.)])^ p(y) = S[y]. 

For other parameter choices, suboptimal ferromagnetic states with < m < 1 can 
also be found by solving the saddle-point equations (jl^) numerically. 

The paramagnetic and ferromagnetic free energies can be easily computed by 
inserting (|T5|) and (|1^) in (|Tl]) to give /para = In 2 — a In ( 2 cosh F^) and 
/ferro = ~ (1 ~ 2/) F^ respectively. One can instantly obtain a phase transition 
occurring at the critical code rate for the BSC Rc = 1 — H2{f), that is valid for every 
code construction under the restrictions K,L > 2, Cj > 1 and Di > 1. This is the 
same phase transition as the one described in |T^. The critical code rate saturates the 



channel capacity and therefore Shannon's coding limit. 

It is important to stress that the coding bound can only be attained in the case of 
unbiased messages. For biased messages (F^ ^ 0) the paramagnetic state ([TBI) is not 
a solution for the saddle-point equations (|l^) and the thermodynamical transition can 
only be obtained numerically and must be bellow the Shannon's bound as can be shown 
by a simple upper bound proposed in 0. 

The upper bound is based on the fact that each bit of the syndrome vector 
z = CnV = Cs^ + CnC (mod 2) is a sum (or product, depending on the representation 
adopted) of K message bits with bias p with L noise bits with flip rate /. The 
probability of Zi = +1 is pt{K,L) = 1/2 (1 + (1 - 2p)^{l - 2/)^). The maximum 
information content in the syndrome vector is then MH2{p~^). For the decoding process 
one has MH2{p't) > NH2{p) + MH2U), resuhing in the bound R < H2{pt) - Hiif). 
Shannon's bound is recovered for unbiased patterns = 1/2, while for biased patterns 
the attainable rates must be bellow Shannon's bound as H2{p^) < 1. 

The main question that remains to be addressed is the accessibility of the various 
states by a practical decoding algorithm. In particular, we will focus on the behef 
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propagation decoding process. In this practical scenario the energy landscape may 
be dominated by the basin of attraction of paramagnetic or suboptimal ferromagnetic 
states even when the ferromagnetic state is the global minimum, degrading the practical 
performance of the code. 



4. Statistical physics and belief propagation 



The decoding problem focuses on finding a Bayes-optimal estimate (also known as 
marginal posterior maximiser, MPM) S for the original message, given the code 
structure, the syndrome vector J' and prior probabilities p and /. 

The Bayes-optimal estimator is defined as an estimator that minimises the posterior 
average of some determined loss function. Using the overlap between message and 
estimate as a loss function, the Bayes-optimal estimator that emerges is of the form 



Sj = sgn{Sj)p(^Sj\j) [HI- The task of computing this estimator is usually very difficult 
as no simple form is known for the posterior P{Sj \ J') and an exponential number of 
operations is required. 




Figure 1. Tanner graph representing tlie neighbour liood of a bit node in an irregular 
MN code. Black circles represent checks and white circles represent bits. 



The problem can be solved in practical time scales by applying the belief propagation 
(BP)[^ framework. In this framework, an approximation for the marginal posterior 
probabilities P{Sj \ J') can be computed iteratively in linear time. For that, a graphical 
representation (belief network) for dependencies between check nodes (or evidence 
nodes) and signal nodes can be constructed. By identifying proper substructures in 
the belief network one can write a closed set of equations whose solutions provide 
the approximation to the posterior probabihties. These substructures can be uniquely 
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identified with conditional distributions. For LDPCs these probabihty distributions are: 

P{Sj = S I {JuaeMsU)\tJ.}) (f^j = PiJfia I Sj = S, {JuKytfia}) (17) 

Pin = r I {Jfi^eMADv}) rl]^ = P{Jf.. \ri = r, {X.^^J), (18) 

where M.s{i) \ (-^n(0 \ ^) denote the set of checks connected to the signal bit j 
(noise bit /) excluding the check containing the bits in /i (noise bits in a). Using Bayes' 
theorem, the posterior probabilities V{Sj \ J) can then be written in terms of g^^^ and 
a prion distributions Po{S) fH). 

The Gibbs weight appearing in Equation (|^, as observed in p2|, |2^, is proportional 
to P{J I S)Pq{S) and can be used to write update formulas for the distributions. 



,(+1) 

ul - 

in p8| one can find the following set of equations: 



Introducing m 



QpA^^ — gL"'^'* and m 



J-i) 

ul 



following the steps described 



rrX 



tanh 



atanh(m^;) + Fs 

y&Ms{l)\ii 



m 



J. n 

teCs{fi)\j 



m 



fit 



n 



m 



fll 5 



(19) 



leCn(p) 



m 



tanh 



atanh(m]^;) 

yeMn{l)\cr 



+ F„ m;^=Jfi n n (20) 

ieCsifl) lGCn{fl)\j 

where the set of signal bits (noise bits) in a check /i (cr) is represented by £s(/i) (£„(/u)). 
The notation £s(/i) \ / indicates all bits in check fj, excluding bit /, Greek letters run 
from 1 to M and Latin letters run from 1 to A^. 

The estimate for the message is Sj = sgn(m^), where is computed as: 



tanh atanh(m^j) + (21) 

The BP decoding dynamics consists of updating Equations (|T9|) and ( pO|) until a 
certain halting criteria is reached, and then computing the estimate for the message 
using equation (^Tj) . The initial conditions are set to reflect the prior knowledge about 
the message ?T^^j(0) = 1 — 2p and noise m";(0) = 1 — 2/. 

The BP algorithm is known to provide the exact posterior when the Tanner graph 
and references therein) associated to the system has a tree architecture 



see 



29 



A 

Tanner graph is a bipartite graph where checks are represented by black circles, bits are 
represented by white circles and an edge connects bits to their related checks. 

When very sparse matrices are used, the probability for a loop in the related graph 
in a finite number of generations decays as where 7 ~ C>{1) For finite 

systems one can expect that a limited neighbourhood of node has a tree structure. 
When applying the thermodynamical limit N ^ 00, the topology actually converges to 
a tree and BP equations become exact. In Figure we show a Tanner graph representing 
the neighbourhood of a bit node in a large irregular MN code. 
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Equations ([I9| ) and pO|) can also be obtained by looking for extrema of the TAP 
free-energy ||18|: 



/TAp(m,m) =-ln2 + -^^ \n {l + m^^m^;) + - J2 J2 1^ (l + ^m.^m.) 



M 



N 



1 



n 



m 



n 



m: 



iV 



n 



1 + ml, + e 



1 



M 



AT ^ 



n (1 



m 



+ e" 



n (1- 

" n (i- 

^leMnU) 



m 



fit 



m 



(22) 



Observe that the TAP free energy described above is not equivalent to the variational 
mean-field free energy introduced in |]ID|, . Here no essential correlations except those 
related to the presence of loops are disregarded. 

The meaning of the fields introduced in the previous section can be understood by 
first applying the gauge transformations m^^- ^-^ ^ji^^liji ^^fij ^ ^ Clival 

and m"o-/ ^ Cifn^ai to the TAP free energy and introducing new variables x = m^^j, 
X = y = mJ^i and y = m"'a-i- If x,x,y and y are interpreted as random variables 

generated by the probability distributions 7r,7f,p and p respectively, one recovers the 
replica symmetric free energy (|TTD (see also |T6[] ). 

From the statistical physics point of view, belief propagation is one of many possible 
ways to find minima of the TAP free energy, representing simple iterative fixed point 
maps. The ferromagnetic state, corresponding to perfect decoding is the global minimum 
up to Shannon's limit in the case of unbiased messages (or very close to it in the case 
of biased messages). However, this equations are very sensitive to the presence of local 
minima in the landscape and the convergence to the global minimum is only expected 
if the initial conditions are set up within the basin of attraction of the ferromagnetic 
state, which requires prior knowledge about the message sent what is not the case in 
practical applications. 

In the next sections we will try to address how the free energy landscape changes 
with the parameters. 



5. Error-correction: regular vs. irregular codes 

Irregularity improves the practical performance of a MN code. We now illustrate this for 
the simplest possible irregular constructions with a probability distribution describing 
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f f 



Figure 2. (a) Magnetization as a function of the noise level / for codes with K = L — 3 
and C = 15 with message bias p — 0.3. Analytical RS solutions for the regular code 
are denoted as O and for the irregular code; with Co = 4 and Cg = 30 denoted as □. 
Results are averages over 10 runs of the TAP/BP algorithm in an irregular code of 
size N = 6000 starting from fixed initial conditions (see the text) ; they are plotted as 
• in the rightmost curve for comparison. TAP/BP results for the regular case agree 
with the theoretical solutions and have been omitted to avoid overloading the figure, 
(b) Free energies for the ferromagnetic state (full line) and for the failure state (line 
with o). The transitions observed in (a) are indicated by the dashed lines. Arrows 
indicate the thermodynamical (T) transition, the upper bound (u.b.) of Section 3 and 
Shannon's limit. 

connectivities of the signal matrix Cg chosen to be: 

Vc{C) = {l-0) 6{C -Co) + e 6{C - Ce). (23) 

The mean connectivity is C = (1 — 6^) Co + 6 Ce and Co < C < Ce] bits in a group with 
connectivity Co will be refered as ordinary bits and bits in a group with connectivity Ce 
as elite bits. The noise matrix C„ is chosen to be regular. 

To gain some insight on the effect of irregularity on solving the TAP/BP equations 
(p!9|) and (pOD we performed several runs starting from the fixed initial conditions 

(0) = 1 — 2p and m"; (0) = 1 — 2/ as prescribed in the last section. For comparison 
we also iterated the saddle-point equations (0) obtained in the replica symmetric (RS) 
theory, setting the initial conditions to be vro(a;) = (1— p) (5(x— m^j(O)) + p 5(a;+m^j (0)) 
and poiy) = (1 — /) S{y — fnai{^)) + / '^(l/+"^"z(0))) suggested from the interpretation 
of the fields tc{x) and p{y) in the last section. 

In Figure ^ (a) we show a typical curve for the magnetization as a function of 
the noise level. The RS theory agrees very well with TAP/BP decoding results. The 
addition of irregularity improves the performance considerably. In Figure ^ (b) we show 
the free energies of the two emerging states. The free energy for the ferromagnetic state 
with magnetization m = 1 is shown as a full line, the failure state (in Figure ^ (a) with 
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magnetization m = 0.4) is shown as a line marked with o. The transitions seen in Figure 
(a) are denoted by dashed hnes. It is clear that they are far below the thermodynamical 
(T) transition, indicating that the system becomes trapped in suboptimal states for 
noise levels / between the observed transitions and the thermodynamical transition. 
The thermodynamical transition coincides with the upper bound (u.b.) in Section 3 
and is very close to, but below. Shannon's limit which is shown for comparison. Similar 
behaviour has already been observed in regular MN codes with = 1 in [ffH] . 




5 10 15 20 

decoding iterations 



25 



Figure 3. Magnetization monitored during the TAP/BP decoding process as a 
function of the number of iterations for N — 4000. Elite nodes magnetization is 
represented by A. Ordinary nodes magnetization is represented by y. The overall 
magnetization is represented by O. The long dashed line shows the dynamics of the 
regular code. The constructions employed have parameters K — L = 3, C — 6, 
Ce = 20 and Co = 5. The noise level is / = 0.065 and the message bias is p = 0.3. 



It is instructive to look how the magnetization of elite (mg) and ordinary (mg) nodes 
evolve throughout the iterative decoding process. In Figure ^ we show this dynamics for 
a regular and an irregular code at a noise level where the irregular code converges to the 
ferromagnetic state while the regular code fails (long-dashed lines). One can see that the 
magnetization of ordinary nodes follow that of the regular code in the first iterations, 
elite nodes are then corrected quickly achieving high magnetization values. These highly 
reliable nodes then lead the correction of ordinary nodes (around the fifth iteration), 
producing successful decoding. From the decoding dynamics point of view irregular MN 
codes can be quahtatively regarded as a mixture of low and highly connected regular 
codes where elite nodes can tolerate higher noise levels while ordinary nodes allow for 
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higher code rates. 



6. The spinodal point 

In the last section we gained some insight on how irregularity affects the practical 
performance of codes. The dynamical decoding process shown in Figure ^ only provides 
a qualitative explanation and does not seem to allow some simple analysis. 

A possible alternative is to relate the observation that the system gets trapped 
in suboptimal states (Figure 0) to global properties of the free energy. The TAP/BP 
algorithm can be regarded as an iterative solution of fixed point equations for the TAP 
free energy (p2D, which is sensitive to the presence of local minima in the system. One can 
expect convergence to the global minimum of the free energy from all initial conditions 
when there is a single minimum or when the landscape is dominated by the basin of 
attraction of this minimum when random initial conditions are used. 

To analyse this point we rerun the decoding experiments starting from initial 
conditions m^j(O) and m";(0) that are random perturbations of the ferromagnetic 
solution : 



mUO) = (1 - Ps) S{mUO) - Q + Ps 5K,(0) + (24) 



and 

m^,(0) = (1 - p„) 6{m:,{0) - n) + p„ 5(m^,(0) + n), (25) 

where for convenience we choose 0<ps = p„ = p<0.5. 

We performed TAP/BP decoding several times for different values of p and noise 
level /. For p < 0.026 we observed that the system converges to the ferromagnetic 
state for all constructions, message biases p and noise levels / examined. It implies that 
this state is always stable. The convergence occurs for any p for noise levels below the 
transition observed in practice. 

These observations suggest that the ferromagnetic basin of attraction dominates the 
landscape up to some noise level fg. The fact that no other solution is ever observed in 
this region suggests that fs is the noise level where suboptimal solutions actually appear, 
namely, it is the noise level that corresponds to the spinodal point of the system. This 
behaviour have already been observed for regular MN codes with K = 1otK = L = 2 
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In ||17| , |T8| we have also shown that MN codes can be divided into three categories 



with different equilibrium properties: [i) K > 3 or L > 3, (ii) K>1, K = L = 2 
and (iii) general L, K = 1. In the next two subsections we will discuss these groups 
separately. 
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Figure 4. Spinodal point noise level fs for regular and irregular codes. In both 
constructions parameters are set as K = L — 3. Irregular codes with Co = 4 and 
Ce = 30 are used. TAP/BP decoding is carried out with N = 5000 and a maximum of 
500 iterations; they are denoted by + (regular) and * (irregular). Numerical solutions 
for the RS saddle-point equations are denoted by O (regular) and O (irregular). 
Shannon's limit is represented by a full line and the upper bound in Section 3 is 
represented by a dashed line. The symbols are chosen to be larger than the actual 
error bars. 

6.1. Biased coding: K > 3 or L > 3 

To show how irregularity affects codes with this choice of parameters we choose K,L = 3, 
Co = 4:, Ce = 30 and biased messages with p = 0.3. These choices are arbitrary but 
can illustrate what happens with the practical decoding performance. In Figure ^ we 
show the transition from the decoding phase to the failure phase as a function of the 
noise level / for several rates R in both regular and irregular codes. Practical decoding 
(O and o) results are obtained for systems of size = 5000 with a maximum number 
of iterations set to 500. Random initial conditions are chosen and the whole process 
repeated 20 times. The practical transition point is found when the number of failures 
equals the number of successes. 

These experiments were compared with theoretical values for fs obtained by solving 
the RS saddle-point equations (|T2|) (represented as + and * in Figure |[) and finding 
the noise level for which a second solution appears. For comparison the coding limit is 
represented in the same figure by a full line. 

As the constructions used are chosen arbitrarily one can expect that these transitions 
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Figure 5. Spinodal point fs for irregular codes as a function of the message bias p. 
The construction is parametrised by X = L = 3, Co = 4 and Ce = 30 with C = 15. 
TAP/BP decoding is carried out with N = 5000 and a maximum of 500 iterations, 
and is represented by +, while theoretical RS solutions are represented by O. The full 
line indicates Shannon's limit. Symbols are larger than the actual error bars 

can be further improved, even though the improvement shown in Figure ^ is aheady 
fairly significant. 

The analytical solution obtained in Section ^ for K>3otL>3,K> 1 and 
unbiased messages p = 1/2 implies that the system is bistable for arbitrary code 
constructions when these parameters are chosen. The spinodal point noise level is then 
/s = in this case and cannot be improved by adding irregularity to the construction. 
Up to the noise level fc the ferromagnetic solution is the global minimum of the free 
energy, and therefore Shannon's limit is potentially saturated, however, the bistability 
makes these constructions unsuitable for practical decoding with a TAP/BP algorithm 
when unbiased messages are considered. 

The situation improves when biased messages are used. Fixing the matrices Cn 
and Cs one can determine how the spinodal point noise level fs depends on the bias 
p. In Figure ^ we compare simulation results with the theoretical predictions of fs as a 
function of p. The spinodal point noise level fs collapses to zero as p increases towards 
the unbiased case. It obviously suggests the use of biased messages for practical use of 
MN codes with parameters i^>3orL>3, -ft'>l under TAP/BP decoding. 

For biased messages with i^>3orL>3,-R'>l the qualitative picture of the 
energy landscape differs from the unbiased coding presented in [0, 0. In Figure ^ 
this landscape is sketched as a function of the noise level / for a given bias. Up to 
the spinodal point fs the landscape is totally dominated by the ferromagnetic state F. 
At the spinodal point another suboptimal state F' emerges, dominating the decoding 
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Figure 6. Pictorial representation of the free energy landscape as a function of the 
noise level /. Up to the spinodal point fs there is only the ferromagnetic state F. At 
fg another state F' appears dominating the decoding dynamics. The thermodynamical 
critical noise level fc indicates the point where the state F' becomes the global 
minimum. 



dynamics. At fc the suboptimal state F' becomes the global minimum. The bold 
horizontal line represents the region where the ferromagnetic solution with m = 1 
dominates the decoding dynamics. In the region represented by the dashed line decoding 
dynamics is dominated by suboptimal m < 1 solutions. 



6.2. Unbiased coding 

For the remaining parameter choices, namely general L, K = 1 and K = L = 2, it was 
shown in |1T7| , |18| that unbiased coding is generally possible yielding close to Shannon's 



limit performance. The free energy landscape of the K = 1 was shown to behave in a 
similar way to the one depicted in Figure H while the landscape of the case K = L = 2 
and unbiased messages shows a different behaviour where some regions include three 
stable states plus their mirror symmetries. 

In the same way as in the K > 3 case the practical performance is defined by the 
spinodal point noise level fs- The addition of irregularity also changes fs in these cases. 

In the general L, K = 1 family we illustrate the effect of irregularity by the choice of 
L = 2, Co = 4: and Cg = 10. In Figure ^we show the transitions observed by performing 
20 decoding experiments with messages of length = 5000 and a maximal number of 
iterations set to 500 (+ for regular and * for irregular). We compare the experimental 
results with theoretical predictions based on the RS saddle-point equations (|T2|) {O 
for regular and o for irregular). Shannon's limit is represented by a full line. The 
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Figure 7. Spinodal point noise level fs for regular and irregular codes. The 
constructions are of K = 1 and L = 2, irregular codes are parametrised by Co = 4 
and Ce = 10. TAP/BP decoding is carried out with N = 5000 and a maximum of 
500 iterations ; they are denoted by + (regular) and * (irregular). Numerical solutions 
for RS equations are denoted by O (regular) and o (irregular). The coding limit is 
represented by a line. Symbols are larger than the actual error bars. 



improvement is modest, what is expected since regular codes already present close to 
optimal performance. Discrepancies between the theoretical and numerical results are 
due to finite size effects. 

We also performed a set of experiments using K = L = 2 with Co = 3 and Ce = 8, 
the same system size = 5000 and maximal number of decoding iterations 500. The 
transitions obtained experimentally and predicted by theory are shown in Figure 

7. Conclusions 

We showed that in the thermodynamic limit MN codes are equivalent to a multi-spin 
ferromagnet submitted to a random field. A replica calculation shows that a phase 
transition from an errorless (ferromagnetic) phase to a failure (either paramagnetic or 
suboptimal ferromagnetic) phase occurs as the noise level increases. The phase transition 
line can be analytically obtained in the case where constructions with K,L > 3, a 
minimum of two checks per bit and unbiased messages {p = 1/2) are used. It coincides 
with Shannon's coding limit and is independent of the code construction. 

For other parameter choices the transition only can be obtained numerically and 
coincides with a simple upper bound, being necessarily below Shannon's limit. 

The practical decoding using belief propagation is shown to attain inferior 
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Figure 8. Spinodal point noise level values fs for regular and irregular codes. 
Constructions are of K = 2 and L ~ 2, irregular codes are parametrised by Co = 3 
and Ce = 8. TAP/BP decoding is carried out with N = 5000 and a maximum of 500 
iterations; they are denoted by + (regular) and * (irregular). Theoretical predictions 
are denoted by O (regular) and o (irregular). The coding limit is represented by a 
line. Symbols are larger than the actual error bars. 

performance to Shannon's limit due to the collapse of the ferromagnetic basin of 
attraction when new states emerge at the spinodal point noise level fs- Irregularity 
increases fs thus improving the code's performance. We show that the maximal noise 
level corrected by an MN code agrees with the replica theory prediction for the spinodal 
point noise level fs- 

This framework is currently being employed for optimising code constructions 
(recently studied in [|l^]), as well as for finding alternatives to the TAP/BP decoding 
scheme and for analysing the effect of using inaccurate priors. 
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